Information processing apparatus, system, and method for detecting collision between a physical and virtual object

ABSTRACT

An information processing apparatus creates a first virtual object expressing a physical object that is detected from physical object information obtained from a physical object information acquisition unit. The information processing apparatus determines a display state of the first virtual object in accordance with a result of detecting collision between the first virtual object and a second virtual object. The information processing apparatus creates, on the basis of a virtual space including the first virtual object and the second virtual object, position-orientation of an HMD, the determined display state, and a physical space image obtained from the HMD, a mixed reality image in combination of an image of the virtual space and the physical space image, and displays the created mixed reality image on the HMD.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of International Patent ApplicationNo. PCT/JP2018/029152, filed Aug. 3, 2018, which claims the benefit ofJapanese Patent Application No. 2017-159101, filed Aug. 22, 2017, bothof which are hereby incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present invention relates to an information processing apparatusdetecting collision between a physical object in a physical space(hereinafter simply referred to as a “physical object”) and a virtualobject in a virtual space (hereinafter simply referred to as a “virtualobject”).

BACKGROUND ART

Research of an MR (Mixed Reality) system is actively conducted aiming torealize seamless connection between a physical space and a virtualspace. In the MR system, an HMD (Head Mount Display), for example, isoften used as a display device. The MR system is utilized in industrialfields as well, for example, to implement a real simulation withoutfabricating a prototype. For example, a layout in a factory can besimulated by superimposing a virtual space of the factory on a physicalspace. On that occasion, in order to perform a simulation such asregarding whether a hand can reach a virtual object in the virtual spacewhen the virtual object is placed somewhere in the factor, it is neededto detect collision between the hand as a physical object and thevirtual object.

When, in a mixed reality space in combination of a physical space and avirtual space, collision between physical objects or between a physicalobject and a virtual object is presented to a user, how both the objectscollide with each other needs to be visualized. In some cases, however,the visualization makes the physical object not recognizable by theuser. To cope with such a problem, Japanese Patent Laid-Open No.2017-033299 discloses a technique of, when virtual objects collide witheach other, leaving a collision trace to clearly indicate the collision.

However, the technique disclosed in Japanese Patent Laid-Open No.2017-033299 is targeted for the collision between the virtual objects,and it does not discuss a point of presenting, to the user, thecollision between the physical objects or between the physical objectand the virtual object. A main object of the present invention is toprovide an information processing apparatus presenting, to the user, thecollision between the physical objects or between the physical objectand the virtual object.

SUMMARY OF INVENTION

The present invention provides an information processing apparatusincluding captured image acquisition means obtaining, on the basis of animage pickup device, a physical space image that is an image of aphysical space, position-orientation information acquisition meansobtaining position-orientation information that representsposition-orientation of a viewpoint of the image pickup device, virtualobject creation means detecting a physical object in the physical spaceand creating a first virtual object that expresses the detected physicalobject, collision detection means detecting collision between the firstvirtual object and a second virtual object, determination meansdetermining a display state of the first virtual object in accordancewith a detection result of the collision detection means, mixed realityimage creation means creating, on the basis of a virtual space includingthe first virtual object and the second virtual object, theposition-orientation information, the display state, and the physicalspace image, a mixed reality image in combination of an image of thevirtual space and the physical space image, and image output meansoutputting the created mixed reality image to be displayed on apredetermined display device.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an MR system.

FIG. 2 is a hardware block diagram of an information processingapparatus.

FIG. 3 is a flowchart representing image processing.

FIG. 4 is an explanatory view illustrating the case in which a hand as aphysical object collides with a virtual object.

DESCRIPTION OF EMBODIMENTS

An embodiment will be described in detail below with reference to thedrawings. The following embodiment discloses an example of practicallyimplementing the present invention and represents a practical example ofthe features stated in Claims.

System Configuration

FIG. 1 is a block diagram of an MR system including an informationprocessing apparatus according to this embodiment. The MR systemincludes an information processing apparatus 1000, an HMD 1200 as anexample of a head-mounted type display device, a display device 1300,and a physical object information acquisition unit 1400. The informationprocessing apparatus 1000 is wired or wirelessly connected to each ofthe HMD 1200, the display device 1300, and the physical objectinformation acquisition unit 1400.

The information processing apparatus 1000 detects collision betweenphysical objects or between a physical object and a virtual object inaccordance with physical object information (such as a stereo image or adistance image) that is obtained from the physical object informationacquisition unit 1400 to detect the physical object in a physical space.In accordance with a result of the detection, the information processingapparatus 1000 creates an image of a mixed reality space (i.e., a mixedreality image) in combination of an image of the physical space (i.e., aphysical space image) obtained from the HMD 1200 and an image of avirtual space (i.e., a virtual space image). The information processingapparatus 1000 displays the mixed reality image, created as mentionedabove, on at least one of the HMD 1200 and the display device 1300. Thephysical object information acquisition unit 1400 is constituted by astereo camera or a distance image camera for obtaining physical objectinformation. The display device 1300 is a general FPD (Flat PanelDisplay) or CRT (Cathode Ray Tube). When a user is only a person whowears the HMD 1200, the display device 1300 is not needed. The displaydevice 1300 is used when multiple users need to confirm the same mixedreality image as that presented to the person wearing the HMD 1200.

HMD

The HMD 1200 includes an image pickup unit 1210R for a right eye, animage pickup unit 1210L for a left eye, an image display unit 1220R forthe right eye, and an image display unit 1220L for the left eye. Theimage pickup units 1210R and 1210L are image pickup devices forcapturing images of the physical space and inputting the capturedimages, as the physical space images, to the information processingapparatus 1000. In a state in which the user is wearing the HMD 1200 onhis or her head, the image pickup unit 1210R is located at a positionwhere it captures an image over a similar range to the visual range ofthe user's right eye, and the image pickup unit 1210L is located at aposition where it captures an image over a similar range to the visualrange of the user's left eye. Therefore, the physical space imagescaptured by the image pickup units 1210R and 1210L are stereo imagesincluding the visual ranges of the user.

The image display units 1220R and 1220L are each a display deviceconstituted by a liquid crystal screen, etc. In the state in which theuser is wearing the HMD 1200 on his or her head, the image display units1220R and 1220L are arranged on the HMD 1200 such that the image displayunit 1220R is positioned in front of the user's right eye and the imagedisplay unit 1220L is positioned in front of the user's left eye. Imagegiven with parallaxes are displayed on the image display units 1220R and1220L. Thus, the HMD 1200 presents a stereo image to the user. The imagedisplay units 1220R and 1220L display the mixed reality image created bythe information processing apparatus 1000.

As described above, the HMD 1200 in this embodiment is of a videosee-through type displaying, on the image display units 1220R and 1220L,the mixed reality image created on the basis of the physical spaceimages captured by the image pickup units 1210R and 1210L. However, theHMD 1200 may be of an optical see-through type displaying a virtualspace image in a superimposed manner on a display medium that enablesthe physical space to be observed with transmission viewing.

Information Processing Apparatus

The information processing apparatus 1000 functions as a captured imageacquisition portion 1010 and a viewpoint information measurement portion1020 in order to execute processing on the physical space images thatare obtained from the HMD 1200. The information processing apparatus1000 further functions as a physical object detection portion 1030, avirtualization portion 1040, a virtual object holding portion 1060, acollision detection portion 1070, and a display switching portion 1080in order to execute processing on the physical object information thatis obtained from the physical object information acquisition unit 1400.Moreover, the information processing apparatus 1000 functions as a mixedreality space creation portion 1090 and an image output portion 1100 inorder to display the mixed reality image on at least one of the HMD 1200and the display device 1300.

The above-described functions may be realized with hardware, but theyare realized with software in this embodiment by executing computerprograms. To that end, the information processing apparatus 1000 has ahardware configuration illustrated in FIG. 2 . The informationprocessing apparatus 1000 includes a CPU (Central Processing Unit) 2010,a ROM (Read Only Memory) 2020, and a RAM (Random Access Memory) 2030.The information processing apparatus 1000 further includes an inputinterface (I/F) 2040 and an output interface (I/F) 2050. The CPU 2010,the ROM 2020, the RAM 2030, the input I/F 2040, and the output I/F 2050are interconnected via a bus 2000 in a communicable manner.

The CPU 2010 controls operation of the information processing apparatus1000 by executing the computer programs that are read from the ROM 2020,while the RAM 2030 is used as a working area. An operating system (OS),various processing programs related to this embodiment, device drivers,etc. are stored in the ROM 2020 and are executed by the CPU 2010 afterbeing temporarily stored in the RAM 2030. The input I/F 2040 isconnected to the HMD 1200 and the physical object informationacquisition unit 1400, and it obtains signals representing images,information, etc., which are in formats processable by the informationprocessing apparatus 1000, from the HMD 1200 and the physical objectinformation acquisition unit 1400. The output I/F 2050 is connected tothe HMD 1200 and the display device 1300, and it outputs signalsrepresenting images that are in formats processable by the HMD 1200 andthe display device 1300. The various functions of the informationprocessing apparatus 1000 will be described below.

The captured image acquisition portion 1010 obtains, from the HMD 1200,the physical space images captured by the image pickup units 1210R and1210L. The captured image acquisition portion 1010 sends the obtainedphysical space images to the viewpoint information measurement portion1020 and the mixed reality space creation portion 1090.

The viewpoint information measurement portion 1020 executes imageprocessing of the physical space images obtained from the captured imageacquisition portion 1010 and extracts information (feature information)representing features in each image, such as dots, lines, and markers.The viewpoint information measurement portion 1020 measuresposition-orientation of a viewpoint by making a position in the image ofthe feature information correspondent to a layout of a space prepared inadvance. The viewpoint information measurement portion 1020 sendsposition-orientation information representing a result of themeasurement to the mixed reality space creation portion 1090. Theposition-orientation information represents a position of the HMD 1200and a direction in which the HMD 1200 is oriented (i.e., a gazedirection). Thus, the viewpoint information measurement portion 1020 hasthe function of obtaining the position-orientation information.

While the position-orientation is measured by executing image processingof a visible image in this embodiment, the position-orientation of thephysical object may be obtained by using, for example, an image capturedwith infrared light or a result detected by an ultrasonic or magneticsensor. Alternatively, the position-orientation of the physical objectmay be obtained by using a distance image captured with a distancesensor, or may be measured in a mechanical way.

The physical object detection portion 1030 obtains the physical objectinformation from the physical object information acquisition unit 1400.The physical object detection portion 1030 detects the physical objectin the physical space from the obtained physical object information andsends a detected result and the physical object information to thevirtualization portion 1040.

The virtualization portion 1040 creates a virtual object expressing thephysical object in the physical space on the basis of the detectedresult of the physical object and the physical object information bothobtained from the physical object detection portion 1030, thusvirtualizing the physical object in the physical space to the virtualobject in the virtual space. The virtualization portion 1040 makescontrol to hold information about a shape of the created virtual object,position-orientation thereof in the virtual space, etc. in the virtualobject holding portion 1060. The virtualization of the physical objectin the physical space to the virtual object in the physical space can beperformed by using KinectFusion (registered trademark) available fromSurreal Vision Ltd., for example. KinectFusion implies real-timethree-dimensional reconstruction using Kinect (registered trademark).When the virtualization to the virtual object is performed by usingKinectFusion, a distance image camera is used as the physical objectinformation acquisition unit 1400. The virtualization portion 1040 firstproduces a group of dots on a physical object surface from a distanceimage that is obtained as the physical object information from thedistance image camera. Then, the virtualization portion 1040 performsthe three-dimensional reconstruction through steps of estimatingposition-orientation of the camera based on ICP (Iterative ClosestPoint), creating a three-dimensional object, and executing rendering.The virtualization portion 1040 may perform the virtualization of thephysical space to the virtual object (i.e., the creation of the virtualobject) by a method other than KinectFusion.

The virtual object holding portion 1060 holds information about thevirtual object constituting the virtual space (such as the shape of thevirtual object and the position-orientation thereof in the virtualspace), information about a light source illuminating the virtual space(such as the position-orientation of the light source), and informationabout the virtual space. The virtual object holding portion 1060 furtherholds not only information about the virtual object created by thevirtualization portion 1040, but also information about a virtual objectcreated with execution of a computer program. Moreover, the virtualobject holding portion 1060 holds information about multiple virtualobjects that are created from multiple physical objects by thevirtualization portion 1040. Those various types of information held inthe virtual object holding portion 1060 are read by the collisiondetection portion 1070 and the mixed reality space creation portion1090.

The collision detection portion 1070 detects collision between thevirtual objects and sends a result of the detection to the displayswitching portion 1080. Because a technique of detecting the collisionbetween the virtual objects is known, description of the technique isomitted.

The display switching portion 1080 determines a display state of thevirtual object, which has been virtualized from the physical objectinformation, in accordance with the result of detecting the collisionbetween the virtual objects. When the virtual object virtualized fromthe physical object information collides with another virtual object,the display switching portion 1080 sets the relevant virtual object tobe displayed (visualized). When both the objects do not collide witheach other, the display switching portion 1080 sets the relevant virtualobject to be not displayed (visualized). The display switching portion1080 makes control to hold, in the virtual object holding portion 1060,information representing the display state of the virtual object in linkwith the information about the virtual object. While it is assumed inthis embodiment to display the virtual object in single color, thevirtual object may be visualized, for example, by displaying it as awireframe, in a blinking manner, in a translucent form, or withgradations depending on distances from a collision position.

The mixed reality space creation portion 1090 creates a virtual spaceimage on the basis of both the virtual object held in the virtual objectholding portion 1060 and the viewpoint position-orientation informationgiven as the result of the measurement by the viewpoint informationmeasurement portion 1020. The virtual space image created here is animage obtained when the virtual space including the virtual object heldin the virtual object holding portion 1060 is viewed from theposition-orientation of the HMD 1200, which is represented by theposition-orientation information. In other words, the virtual spaceimage is given as an image captured under conditions of the sameorientation and range as those of the physical space image. The mixedreality space creation portion 1090 executes creation of an image of themixed reality space (i.e., creation of the mixed reality image) bysuperimposing the physical space image obtained from the captured imageacquisition portion 1010 on the created virtual space image. The mixedreality space creation portion 1090 sends the created mixed realityimage to the image output portion 1100. Because a technique of creatinga virtual space obtained when viewing the virtual object from apredetermined position represented by the position-orientationinformation is known, detailed description of the technique is omitted.

The image output portion 1100 sends the mixed reality image obtainedfrom the mixed reality space creation portion 1090 to at least one ofthe HMD 1200 and the display device 1300. The HMD 1200 displays themixed reality image obtained from the image output portion 1100 by theimage display units 1220R and 1220L. The display device 1300 alsodisplays the mixed reality image obtained from the image output portion1100.

Processing

FIG. 3 is a flowchart representing image processing that includes thephysical-object collision detection in the MR system described above. Inthe image processing, the mixed reality image is created and displayed.Here, the physical object is assumed to be a user's hand, for example,and whether the hand collides with the virtual object is detected. As amatter of course, the physical object is not limited to the hand.

In the information processing apparatus 1000, the physical objectdetection portion 1030 obtains the physical object information about theuser's hand from the physical object information acquisition unit 1400(S3000). The virtualization portion 1040 obtains the physical objectinformation about the hand from the physical object detection portion1030 and creates a virtual object (polygon) of the hand from thephysical object information (S3100). The hand in the physical space isthereby virtualized to a virtual object. The hand polygon is held in thevirtual object holding portion 1060. The display switching portion 1080sets the hand polygon held in the virtual object holding portion 1060 tobe not displayed (S3200).

The collision detection portion 1070 detects whether the hand polygonheld in the virtual object holding portion 1060 collides with anothervirtual object (S3300). If a detection result shows that the handpolygon collides with another virtual object (S3400: Y), the displayswitching portion 1080 sets the hand polygon to be displayed (S3500). Ifthe detected result shows that the hand polygon does not collide withanother virtual object (S3400: N), the display switching portion 1080sets the hand polygon to be not displayed (S3600).

The mixed reality space creation portion 1090 creates the mixed realityimage on the basis of the virtual object held in the virtual objectholding portion 1060, the position-orientation information that is theresult of the measurement by the viewpoint information measurementportion 1020, and the physical space image obtained from the capturedimage acquisition portion 1010 (S3700). The image output portion 1100outputs the created mixed reality image to the HMD 1200 (S3800). Theinformation processing apparatus 1000 determines whether the processingis to be ended after outputting the mixed reality image (S3900). If theprocessing is not to be ended (S3900: N), the information processingapparatus 1000 repeatedly executes the processing in S3000 and thesubsequent steps. If the processing is to be ended (S3900: Y), theinformation processing apparatus 1000 ends the processing at once.

With the above-described information processing apparatus 1000 accordingto this embodiment, the physical object is virtualized, and an image ofthe virtualized physical object is visualized only when the virtualizedphysical object collides with another virtual object. Therefore, thecollision between the physical object and the virtual object can bepresented to the user without disturbing the image in the real world,which is captured by the HMD 1200. Furthermore, when the other virtualobject is also an image of a virtualized physical object, collisionbetween the physical objects can be presented to the user.

The virtual object having been visualized because of collision withanother virtual object may be continuously visualized even after thecollision state has been cleared. In such a case, control may beperformed such that a relative position of the relevant virtual objectto the other virtual object in collision with the former is keptunchanged from the position at a time when the collision has occurred.This can be realized by holding the relative position at the time whenthe collision has occurred, and by drawing a virtual object to belocated at the relative position with respect to the other virtualobject as a reference. FIG. 4 is an explanatory view illustrating thecase in which a hand as a physical object collides with a virtualobject. The hand 4020 as the physical object is virtualized to a virtualobject.

When another virtual object 4010 and the virtual object of the hand 4020do not collide with each other, a virtualized hand 4030 is notdisplayed, and the user can view only the hand 4020 as the physicalobject. When the other virtual object 4010 and the virtual object of thehand 4020 collide with each other, the virtualized hand 4030 isvisualized and displayed. At that time, a relative position of thevirtualized hand 4030 to the other virtual object 4010 is fixed to aconstant position.

Since the relative position of the virtualized hand 4030 to the othervirtual object 4010 is fixed to the constant position in the state inwhich the virtualized hand 4030 is visualized, a collision conditionbetween the physical object and the virtual object (i.e., how both theobjects collide with each other) can be presented to the user.

In the above description, the image processing including thephysical-object collision detection is executed by one unit of theinformation processing apparatus 1000. However, the image processingincluding the physical-object collision detection may be executed by asystem including multiple units of the information processing apparatus1000. In other words, each of the multiple information processingapparatuses may execute the collision detection. Collision detectionresults of the individual information processing apparatuses may becollected into any one of the information processing apparatuses, andthe one information processing apparatus may determine, withcomprehensive judgment based on those collision detection results,whether the virtualized physical object is to be visualized. A manner ofexpressing the virtual object may be changed depending on the collisiondetection results of the individual information processing apparatuses.For example, a luminance, a color, a contour, etc. of the virtual objectmay be changed depending on a collision time and a collision area, forexample.

The present invention can also be implemented with a process ofsupplying one or more programs for realizing one or more of thefunctions of the above-described embodiment to a system or a device viaa network or a storage medium, and causing one or more processors in thesystem or the device to read the one or more programs and to execute theread program(s). Alternatively, the present invention can be furtherimplemented by a circuit (e.g., ASIC) realizing the one or morefunctions.

Any of the above-described embodiments represents a practical example inimplementing the present invention, and the technical scope of thepresent invention is not to be construed in a restrictive sense by theabove-described embodiments. In other words, the present invention canbe implemented in various forms without departing from the technicalscope or the main features of the present invention.

According to the present invention, the collision between the physicalobjects or between the physical object and the virtual object can bepresented to a user.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

The invention claimed is:
 1. An information processing apparatuscomprising: one or more processors; and one or more memories coupled tothe one or more processors, the one or more memories having storedthereon instructions which, when executed by the one or more processors,cause the apparatus to: obtain, from an image pickup device, a physicalspace image that is an image of a physical space captured by the imagepickup device; obtain position and orientation information thatrepresents a position and an orientation of a viewpoint of the imagepickup device; detect a physical object in the physical space; create afirst virtual object that expresses a three dimensional structure of thedetected physical object; set a display state of the first virtualobject to an invisible state in a case where the first virtual objectand a second virtual object do not collide; detect collision between thefirst virtual object and the second virtual object; change an appearanceof the first virtual object so as to inform a user of the collisionbetween the detected physical object and the second virtual object byswitching the display state of the first virtual object from theinvisible state to a visible state in response to the detection of thecollision; create, based on a virtual space including the first virtualobject and the second virtual object, the position and orientationinformation, the display state, and the physical space image, a mixedreality image in combination of an image of the virtual space and thephysical space image; and output the created mixed reality image to bedisplayed on a display device, wherein the mixed reality image iscreated so that a relative position at the time of the collision betweenthe first virtual object in the visible state and the second virtualobject is maintained even when the detected physical objectcorresponding to the first virtual object is separated from the secondvirtual object after the collision.
 2. The information processingapparatus according to claim 1, wherein in the displayed state, thefirst virtual object is displayed as a wireframe, in a blinking manner,or in a translucent form when the first virtual object and the secondvirtual object collide with each other.
 3. The information processingapparatus according to claim 1, wherein in the displayed state, thefirst virtual object is displayed with gradations depending on distancesfrom a collision position when the first virtual object and the secondvirtual object collide with each other.
 4. The information processingapparatus according to claim 3, wherein the second virtual object isobtained by creating a virtual object that expresses another physicalobject.
 5. The information processing apparatus according to claim 4,wherein from measurement device measuring the position and orientationof the viewpoint of the image pickup device, the position andorientation information that represents the position and orientation ofthe viewpoint of the image pickup device is created.
 6. The informationprocessing apparatus according to claim 5, wherein the measurementdevice measures the position and orientation of the viewpoint of theimage pickup device based on features detected from an image picked upby the image pickup device.
 7. A system comprising a plurality of theinformation processing apparatuses according to claim 1, wherein each ofthe plurality of the information processing apparatuses detects thecollision between the first virtual object and the second virtualobject; any one of the information processing apparatuses determines adisplay state of the first virtual object with comprehensive judgmentbased on collision detection results in the individual informationprocessing apparatuses.
 8. The system according to claim 7, wherein theany one of the information processing apparatuses changes a manner ofexpressing the first virtual object in accordance with the collisiondetection results in the individual information processing apparatuses.9. An image processing method executed by an information processingapparatus connected to an image pickup device capturing a physical spaceimage that is an image of a physical space, and to a display device, theimage processing method comprising steps of: obtaining position andorientation information that represents a position and an orientation ofa viewpoint of the image pickup device; detecting a physical object inthe physical space; creating a first virtual object that expresses athree dimensional structure of the detected physical object; set adisplay state of the first virtual object to an invisible state in acase where the first virtual object and a second virtual object do notcollide; detecting collision between the first virtual object and thesecond virtual object; change an appearance of the first virtual objectso as to informing a user of the collision between the detected physicalobject and the second virtual object by switching the display state ofthe first virtual object from the invisible state to a visible state inresponse to the detection of the collision; and creating, based on avirtual space including the first virtual object and the second virtualobject, the position and orientation information, the display state, andthe physical space image, a mixed reality image in combination of animage of the virtual space and the physical space image; and outputtingthe created mixed reality image to be displayed on the display device,wherein the mixed reality image is created so that a relative positionat the time of the collision between the first virtual object in thevisible state and the second virtual object is maintained even when thedetected physical object corresponding to the first virtual object isseparated from the second virtual object after the collision.
 10. Anon-transitory storage medium storing a computer program for a computerconnected to an image pickup device capturing a physical space imagethat is an image of a physical space, and to a display device, theprogram causing the computer to execute steps of: obtaining position andorientation information that represents a position and an orientation ofa viewpoint of the image pickup device; detecting a physical object inthe physical space; creating a first virtual object that expresses athree dimensional structure of the detected physical object; set adisplay state of the first virtual object to an invisible state in acase where the first virtual object and a second virtual object do notcollide; detecting collision between the first virtual object and thesecond virtual object; change an appearance of the first virtual objectso as to informing a user of the collision between the detected physicalobject and the second virtual object by switching the display state ofthe first virtual object from the invisible state to a visible state inresponse to the detection of the collision; and creating, based on avirtual space including the first virtual object and the second virtualobject, the position and orientation information, the display state, andthe physical space image, a mixed reality image in combination of animage of the virtual space and the physical space image; and outputtingthe created mixed reality image to be displayed on the display device,wherein the mixed reality image is created so that a relative positionat the time of the collision between the first virtual object in thevisible state and the second virtual object is maintained even when thedetected physical object corresponding to the first virtual object isseparated from the second virtual object after the collision.